Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combining PRs for streamlining testing and merging #2430

Closed
wants to merge 84 commits into from

Conversation

aerorahul
Copy link
Contributor

@aerorahul aerorahul commented Mar 24, 2024

DO NOT MERGE

This PR is not running cleanly without issues on Hera.

Description

This PR combines:

Note the TODO for installing fit2obs version 1.1.0 on Orion, Hercules and WCOSS2 and updating run.spack.ver etc.

Type of change

  • Bug fix (fixes something broken)
  • New feature (adds functionality)
  • Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO

How has this been tested?

Since Jenkins on Hera is not available due to Java issues, a bunch of CI runs are being run on Hera at:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/

Update on tests:

The C48mx500_3DVarAOWCDA test fails in gdasprepoceanobs for a missing file gdas.t18z.sst_avhrr_ma_l3u.2021032418.nc4.
See log: /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032418/gdasprepoceanobs.log
In the same experiment the gdasanal job failed with the error:

82: [082]correlated_obsmod*upd_varch_(): >>> ERROR <<<  improperly set GSI_BundleErrorCov
82: [082]dropdead(): at 20240324:003122.203(z-000000)

See log:
/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C48mx500_3DVarAOWCDA/logs/2021032418/gdasanal.log
The analysis jobs for 3DVar and 4DEnVar run to completion.

In the C96_atmaerosnow test job prepsnowobs.sh, pip is not found in load_ufsda_modules.sh

/scratch1/NCEPDEV/global/Rahul.Mahajan/gwWork.rocky8/consol/ush/load_ufsda_modules.sh: line 52: pip: command not found

See log:
/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96_atmaerosnowDA/logs/2021122018/gdasprepsnowobs.log
pip list was taken out from load_ufsda_modules.sh

prepsnowobs.sh has an incorrect path to iodaconv. This is corrected.
pyiodaconv is no longer built w/ python 3.7. It is hard-wired, and now updated to 3.10.

Offline UPP job for atm analysis atmanlupp as well as ecen jobs are seeing GLIBC errors like below:

+ JGLOBAL_ATMOS_UPP[22]: /scratch1/NCEPDEV/global/Rahul.Mahajan/gwWork.rocky8/consol/scripts/exglobal_atmos_upp.py
python3: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/libpython3.11.so.1.0)
python3: /lib64/libc.so.6: version `GLIBC_2.27' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/python-3.11.6-b6ydksr/lib/libpython3.11.so.1.0)

See log:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96_atm3DVar/logs/2021122100/gdasatmanlupp.log

and

+ exgdas_enkf_ecen.sh[186]: srun -l --export=ALL -n 80 --cpus-per-task=2 /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNDIRS/C96C48_hybatmDA/ecen.190800/getsigensmeanp_smooth.x ./ atminc_ensmean atminc 2
 1: /scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNDIRS/C96C48_hybatmDA/ecen.190800/getsigensmeanp_smooth.x: /lib64/libc.so.6: version `GLIBC_2.28' not found (required by /scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/gsi-addon-dev-rocky8/install/intel/2021.5.0/hdf5-1.14.0-lixiejp/lib/libhdf5.so.310)

The log:

/scratch1/NCEPDEV/stmp2/Rahul.Mahajan/RUNTESTS/pr2430_6218b3/COMROOT/C96C48_hybatmDA/logs/2021122100/enkfgdasecen001.log

These could these be related to stack or the machine, but several other python jobs are running fine as well as other executables. A Hera helpdesk ticket has been opened.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

aerorahul and others added 30 commits February 6, 2024 12:02
DavidHuber-NOAA and others added 29 commits March 21, 2024 14:51
Co-authored-by: Rahul Mahajan <[email protected]>
The new Fit2Obs tag is v1.1.0. Change fit2obs_ver=1.1.0

Refs NOAA-EMC#2329
Set the Fit2Obs MODULEPATH back to use fit2obs_ver.

Refs NOAA-EMC#2329
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants